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Abstract 

The  goal  of  universal  source  coding  Is  to  find  codes  which  are 
efficient  for  sources  with  unknown  or  Incompletely  specified  probability 
distributions;  this  Is  equivalent  to  finding  codes  which  are  efficient 
for  a collection  of  sources.  Previous  work  In  universal  variable-rate 
source  coding  subject  to  a fidelity  constraint  has  assumed  a prior 
probability  measure  on  a collection  of  stationary  ergodlc  sources.  It  Is 
then  shown  that  there  Is  a single  code  which  gives  an  average  distortion  D 
at  an  optlmxim  rate,  Rg(D),  for  most  sources  9 In  the  collection.  This  code 
Is  necessarily  variable-rate  since  H0(D)  Is  not  constant  over  the  collection 
of  sources. 

In  this  dissertation,  new  results  are  presented  on  coding  a 
collection  of  stationary  ergodlc  sources  when  no  prior  probability  measure 
Is  assumed.  In  particular,  It  Is  shown  that  there  exist  variable-rate 
universal  codes  whose  rate-distortion  performance  converges  to  the  optimum 
attainable  performance  for  each  source  In  the  collection.  Two  types  of 
universal  codes  are  considered  In  which  the  above  convergence  Is  polntwlse 
(weakly-universal)  or  uniform  (strongly-unlversal)  over  the  collection  of 
sources.  A major  result  of  the  work  is  that  weakly-universal  codes  exist 
for  a collection  of  stationary  ergodlc  sources  with  source  and  reproducing 
alphabets  separable  metric  spaces  and  the  distortion  measure  a nonnegative 


nondecreasing  function  of  the  metric.  It  Is  assumed  that  there  Is  a letter 
In  the  reproducing  alphabet  giving  finite  average  distortion  when  used  with 
each  source  In  the  collection  and  that  sup{Rg(D)}  over  the  collection  of 
sources  Is  finite.  Strongly-unlversal  codes  are  shown  to  exist  for 
collections  of  finite  alphabet  memoryless  sources  and  certain  types  of 
Markov  sources. 


# 


Acknowledgements 


I would  like  to  thank  my  thesis  adviser.  Dr.  M.  B.  Pursley, 
for  his  patience,  support,  encouragement,  and  many  helpful  comments 
during  the  research  period.  I would  like  to  thank  Drs.  A.  H.  Haddad, 

F.  P.  Preparata,  and  R.  B.  Ash  for  serving  on  the  doctoral  examination 
connlttee. 

Many  thanks  go  to  Phyllis  Young  for  her  endurance  and  expertise 
during  the  typing  of  this  manuscript. 


Iv 

Table  of  Contents 

CHAPTER  Page 

1.  Introduction  1 

2.  Mathematical  Preliminaries  7 

3.  Weakly  Universal  Variable  Rate  Codes  10 

4.  Strongly  Universal  Variable  Rate  Codes  16 

Appendix  A 19 

Appendix  B 38 

References  48 

Vita  50 


1 


Chapter  1.  Introduction 

Shannon's  classical  papers  In  Information  theory  [l,2]  deal  with 
the  coding  of  a source  with  known  probability  distribution.  However,  In 
practice,  the  probability  distribution  of  the  source  Is  rarely  (If  ever) 

known  precisely,  and  yet  It  Is  desired  to  find  a rode  which  performs  ^ 

asymptotically  as  well  as  a code  designed  with  full  knowledge  of  the  source 
probability  distribution.  We  will  refer  to  this  as  the  universal  coding 
problem. 

We  assume  In  the  treatment  of  this  problem  that  the  reproducing 
alphabet  used  to  encode  the  source  Is  known  and  that  the  source  alphabet 
Is  a subset  of  some  known  set  A^.  If  nothing  Is  known  about  the  source 
probability  distribution,  then  the  universal  coding  problem  has  no  solution 
since  the  unknown  source  could  be  ergodlc,  stationary,  or  non- stationary 
and  the  universal  code  would  be  required  to  work  well  In  each  of  these 
cases.  Therefore,  It  Is  necessary  to  give  some  Information  about  the 
probability  distribution  of  the  unknown  source;  In  our  discussion  and  the 
great  majority  of  the  literature,  this  Is  done  by  specifying  some  regu- 
larity property  of  the  source,  l.e..  It  Is  stated  that  the  unknown  source 
Is  memoryless,  Markov,  stationary  ergodlc,  stationary,  etc.  With  this  and 
sometimes  additional  Information,  Interesting  results  can  be  obtained  on 
the  universal  coding  problem. 

It  Is  now  possible  to  give  an  equivalent  formulation  of  the 
universal  coding  problem.  The  unknown  source  Is  assumed  to  be  a member  of 
the  class  of  all  sources  having  the  same  regularity  property  and  other 
properties  assumed  for  the  unknown  source;  sources  In  the  class  are  Indexed 
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by  a parameter  6 taking  values  in  a set  A.  The  universal  coding  problem 
Is  then  equivalent  to  the  problem  of  finding  a single  code  which  works 
well  for  each  member  of  the  class  of  sources  with  Index  set  A.  Henceforth, 
we  discard  the  notion  of  an  unknown  source  and  Instead  refer  to  this 
equivalent  formulation  when  discussing  the  universal  coding  problem.  In 
our  discussion,  we  seek  only  universal  codes  which  encode  blocks  of  source 
outputs  Independently,  that  Is,  the  encoding  of  a particular  block  output 
does  not  depend  on  the  source  outputs  during  any  previous  blocks. 

Universal  codes  can  be  roughly  divided  Into  vdiether  the  code 
reproduces  the  source  perfectly  (noiseless  coding)  or  whether  distortion 
Is  allowed  (coding  with  a fidelity  criterion).  We  will  be  concerned  with 
only  the  latter  type  of  code  In  this  irark.  Universal  noiseless  coding  has 
been  discussed  by  Fltlngof  [3],  Zlv  [4],  and  Davisson  [5]. 

There  are  two  distinct  philosophies  In  universal  coding  with  a 
fidelity  criterion.  The  most  prevalent  approach  and  historically  the  first 
has  been  to  use  fixed  length  block  codes  of  some  fixed  rate  R to  achieve 
an  optimum  average  distortion,  D0(R),  for  most  6 € A.  Since  these  codee 
guarantee  a transmission  rate  R for  each  9 € A,  they  can  be  used  to 
transmit  over  a fixed  rate  channel  If  the  user  Is  willing  to  tolerate  the 
large  average  distortions  that  may  result  for  some  6 € A. 

There  are  several  different  categories  of  fixed  rate  universal 
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codes.  If  W Is  any  weighting  measure  assigned  to  the  set  A and  C Is  a 
fixed  length  block  code  of  rate  R whose  average  distortion  for  the  source 
with  Index  9 Is  p(C,9)  then  we  say  C Is  a weighted  universal  fixed  rate 


3 


code  if 

p(C,9)W(de)  pa  DQ(R)W(d0). 

Such  codes  were  first  discussed  by  Gray  and  Davisson  [6] , where  It  was  shown 

A 

that  weighted  universal  codes  exist  for  a separable  metric  space,  a 

A 

totally  bounded  metric  space,  the  distortion  measure  p a metric  on  A^  U A^, 
and  A an  Index  set  for  the  class  of  all  stationary  ergodlc  sources  on  A^. 

If  1*  a stationary  source  with  source  alphabet  A^,  then  the  ergodlc 
decomposition  theorem  [7]  allows  us  to  Identify  W with  p,  and  A with  A^  and 
thereby  show  that  the  weighted  average  distortion  of  the  universal  code  Is 
the  optimum  distortion  attainable  by  any  code  of  the  same  rate  applied  to 
the  stationary  source  [6] . Another  result  which  follows  quickly  from  the 
existence  of  weighted  universal  codes  Is  that  for  any  e > 0,  there  Is  a setj 
Aq  c A with  W(Aq)  ^ 1 - e and  a single  fixed  length  block  code  C of  rate  R 
for  which 

p(C,0)  pa  Dq(R) 

for  all  0 € Aq.  Such  a code  Is  universal  In  the  sense  that  It  gives  an 

optimum  average  distortion  for  "most”  0 € A. 

Two  further  types  of  universal  codes,  weakly  mlnlmax  and  strongly 

mlnlmax.  dlspenae'^lth  the  assumption  of  a prior  weighting  measure  on  the 

class  A [8].  Weakly  mlnlmax  codes  exist  for  A If  there  Is  a sequence  [c^} 

of  fixed  length  block  codes  of  fixed  rate  R such  that  p(C  ,0)  -•  D.(R)  for 

n 9 

each  9 € A.  Strongly  mlnlmax  universal  codes  exist  for  A If  there  Is  a 
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s'^ngle  fixed  length  block  code  C of  rate  R such  that  p(C,0)  D0(R)  for 

all  0 € A.  Zlv  first  showed  the  existence  of  weakly  mlnimax  universal  codes 
for  general  sources  and  alphabets  [9] . His  results  were  generalized  further 
by  Neuhoff,  Gray,  and  Davisson  [8]  who  showed  that  weakly  mlnimax  universal 

A 

codes  exist  for  the  class  of  all  stationary  ergodlc  sources  with  and 

A 

metric  spaces  such  that  either  Aq  or  Aq  Is  separable  and  the  distortion 
measure  p a metric  on  Aq  U Aq,  In  [8] , strongly  mlnimax  universal  codes  were 
shown  to  exist  for  a class  of  sources  totally  bounded  under  the  p distance 
measure  [lO]  If  Aq  Is  finite  or  a totally  bounded  metric  space.  An  example  of 
a class  of  sources  totally  bounded  under  the  p distance  Is  a class  of  memory- 
less sources  on  a metric  space  alphabet  and  a class  of  finite  alphabet  n-th 
order  Markov  sources  having  transition  probabilities  bounded  away  from  zero. 

In  [ll],  Neuhoff  and  Shields  discuss  strongly  mlnimax  coding  for  Markov 
sources.  They  show  that  strongly  mlnimax  codes  exist  for  the  class  of  all 
binary  Ist-order  Markov  sources,  and  that  for  Ist-order  Markov  sources  with 
source  alphabets  of  K symbols,  K > 2,  strongly  mlnimax  codes  of  rate  R exist 
If  and  only  If  R ^ log(K>l)> 

The  basic  motivation  behind  the  second  approach  to  universal  coding 


Is  to  find  universal  codes  which  guarantee  average  distortion  ^ D for  all 
9 € A.  An  efficient  universal  code  which  guarantees  average  distortion  lis  D 
for  each  9 € A should  have  average  rate  at  RgCD)  for  each  0 € A.  It  Is  clear 
that  to  meet  these  requirements  when  R0(D)  is  not  a trivial  function  of  9. 
a variable  rate  coda  must  be  used.  To  accomplish  this,  a binary  Interface 
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Is  Inserted  between  the  source  output  block  and  the  block  of  reproduction 
letters.  The  binary  Interface,  actually  a binary  variable  length  prefix 
condition  code  book.  Is  then  designed  so  that  for  each  9 € A the  encoding 
and  decoding  mapping  preserves  the  distortion  properties  of  a good  block 
code  specifically  designed  for  that  6 and  so  that  the  average  transmission 
rate  of  the  binary  code  Is  close  to  Rg(D).  Pursley  and  Davisson  [12]  first 
took  this  approach  to  universal  coding  and  obtained  results  In  the  variable 
rate  sense  which  roughly  correspond  to  weighted  universal  fixed  rate  results. 

In  the  variable  rate  terminology,  we  say  that  the  class  of  sources  Is 
W-almost  universally  encodable  If  for  each  e > 0 there  exists  a binary 
variable  rate  code  B and  a set  c A with  W(Aq)  s 1 - e such  that  for 
each  0 € A^, 

P(B,0)  s D + e 

and  r(B,0)  s (ln2)"^[Rg(D)  + e] . 

Here  p(B,0)  and  r(B,0)  are  the  average  distortion  and  average  rate  of  B for 
0 € A.  In  [12],  Pursley  and  Davisson  showed  that  the  class  of  all  stationary 

A 

ergodlc  sources  with  a separable  metric  space,  a totally  bounded  metric 
space  and  p a metric  on  A^  U A^  Is  W-almost  universally  encodable.  In  [13], 

A 

this  result  was  extended  to  allow  A^  to  be  either  a Hilbert  space  or  a 
metric  space  for  which  every  closed  bounded  subset  Is  compact.  Subsequently, 

Kleffer  [14]  proved  the  result  for  A^  a separable  metric  space  and  p a 
nonnegative  nondecreasing  function  of  the  underlying  metric. 

I 

! 
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One  potential  use  for  variable  rate  universal  codes  would  be  In 
the  transmission  of  satellite  weather  photographs.  The  weather  Images  are 
often  stored  In  digital  form  In  a computer  memory  or  on  magnetic  tape.  In 
converting  the  original  Image  to  digital  form,  distortion  Is  necessarily 
Introduced.  An  efficient  universal  variable  rate  code  would  guarantee  a 
fixed  average  distortion  In  the  digital  reproduction  process  and  allow  the 
video  Information  to  be  stored  on  the  mlnlmtim  amount  of  magnetic  tape  or 
computer  memory. 

In  this  dissertation,  we  seek  to  obtain  variable  rate  universal 
coding  results  for  general  sources  and  distortion  measures  without  the  use 
of  a prior  weighting  measure  on  A.  We  define  two  types  of  universal  variable 
rate  codes  which  are  analogous  to  fixed  rate  weakly  and  strongly  mlnlmax 
universal  codes.  New  results  on  these  types  of  universal  codes  are  presented 
In  Chapters  3 and  4,  and  In  Appendices  A and  B,  the  proofs  of  results  in 
Chapters  3 and  4,  respectively,  are  presented.  Chapter  2 contains  necessary 
mathematical  details  for  the  subsequent  discussion. 
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Chapter  2.  Mathematical  Preliminaries 


We  let  denote  a a-fleld  of  subsets  of  the  source  alphabet  A^. 

If  there  are  no  other  restrictions  on  such  as  topological  restrictions 
(e.g.,  metrlzablllty  or  separability)  or  restrictions  on  the  cardinality, 
then  we  will  refer  to  A^  as  an  abstract  alphabet.  For  n > m where  n and  m 

n 

are  Integers  or  + define  A ■ x A^  and  (/  - x where  A ■ A„  and 

“ to 

t^  t^ 

(2 . • for  m s t s n-1.  We  write  A for  A*  and  d for  d^  . We  let 

t U ••00  «G0 

00  Q 

denote  the  C7-fleld  of  all  sets  of  the  form  G x A,  where  G € (7  and  for  each 

\ *00 

00 

positive  Integer  n we  let  denote  the  a-fleld  of  all  sets  of  the  form 
a||„^  X G where  G € For  ou  ■ . . . . . . € A and  Integers  t,  define 

X^:  A -•  Aq  by  X^(uj)  ■ u)^.  Let  T be  the  shift  transformation  on  A,  For 
positive  Integers  k,  define  X*^ ; A - A^  by  X*'  ■ (X^.Xj^, . . . ,Xj^_j).  Denote 
elements  of  Aq  by  x”  “ (*0’*1*  * * ’ **n-l^‘  n ■ jk  where  j and  k are 

integers,  let  • (*ik»*ik+l’ * * * ♦’^ik-Ht-l^  for  0 s i s j-i;  i.e., 

(x").  . is  the  (l+l)-st  k vector  of  x”. 

^ 1 

A A 

For  reproduction  space  (Aq,<7q),  let  p : A^  x A^  -•  [O,®)  be  an 
d^  X ^T^-measurable  function,  and  define  p^;  Aq  X Aq  -•  [O,®)  by 

-1 

p (x",^")  ■ n“^  S p(x.  ,y.).  We  will  often  abbreviate  p (x”,;^.")  *»y  P(x"ti*)< 
” *"  1-0  n - 

Let  ^q(*)  he  the  rate  distortion  function  (In  nats)  for  the  stationary 

source  (A/7,^q),  distortion  measure  p,  and  reproduction  alphabet  A q.  Let 

€ a}  denote  an  arbitrary  collection  of  such  sources. 

V 
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If  C is  any  finite  set,  let  |c|  be  the  cardinality  of  C.  Let 
[c]*^  be  the  k»fold  product  set  C x C x ...  C.  If  C c a”,  we  call  C a 
block  code  of  length  n.  For  jc”  € A^,  let 

P^(x"lc)  - min  e C]. 

Following  [12],  a binary  variable-rate  code  on  blocks  of  length  n. 
B^,  is  a triple  (B,U,V),  consisting  of  codebook  B,  a finite  set  of  binary 
sequences  satisfying  the  prefix  condition;  an  (7^-measurable  encoding  mapping 
U:  Aq  -•  B;  and  a decoding  mapping  V:  B -*  A^.  Let  £:  B-{l,2,...}  be  the 
function  which  assigns  to  each  codeword  in  B its  length  in  bits.  The  average 
distortion  for  code  B^ applied  to  the  Q-th  source  is 

Pn^-n‘®  > “ 

and  average  rate  (in  bits  per  source  symbol)  is 

rn(Bn,9)  “ E^n’^f  [u(x")] 

where  Eq  is  the  expectation  with  respect  to  source  (A/7,Uq);  that  is, 

'e*  ■ /a  f '^‘e 

for  any  C7>measurable,  real-valued  function  f . When  the  code  B is  clear  from 

context  we  will  often  denote  p.(B  ,9)  and  r (B  ,6)  by  p (6)  and  r (6). 

n'— n n — n’  ^ ^n  n' 

As  discussed  in  [12],  any  block  code  C of  length  n has  a natural 
representation  as  a binary  variable-rate  code,  which  we  denote  B(C).  In 
this  representation.  If  B(C)  ■ (B,U,V),  then 


1 


I 


I 


I 


1 


1 

n 


I 


! 


V 


Pn(2l".V[U(x")]  ) - p^(x"|c) 


(1) 


and 

-4[u  (x”)]  = riog^lcl"!  (2) 

for  all  X € Aq  where  riog2|c|l  is  the  smallest  Integer  greater  than  or 
equal  log^jcl. 

The  following  statement  of  the  source  coding  theorem  will  be  used 
frequently  in  the  remainder  of  the  dissertation.  This  result  is  proven  in 
[l5]  for  finite  source  alphabets,  but  the  proof  extends  to  abstract  source 
alphabets  using  techniques  of  [16,  Ch.  7],  as  stated  in  [l5]. 

Theorem  1;  Let  (A/7,Hg)  be  a stationary  ergodlc  source  and  assume  there  is 

A 

a letter  b € A^  for  which  Eq  p(XQ,b)  < ®.  Then  for  any  6 > 0 and  any  D 2 0 
for  which  Rg(D)  < ®,  there  is  an  integer  such  that  for  n 2 , there  is  a 

block  code  C of  length  n,  such  that  b"  « (b,b,...,b)  € C, 

^^e^P(x"|c)  > D + 6]  < 6, 

and 

n’^riog^lcp  < (ln2)"^[Rg(D)  +6]. 
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chapter  3.  Weakly  Universal  Variable  Rate  Codes 

In  this  chapter  we  consider  weakly  universal  source  coding  which 
Is  defined  as  follows. 

Definition  1.  A family  )|0  € a}  of  stationary  ergodlc  sources  Is 

weakly  universally  encodable  at  distortion  level  D with  reproduction 
alphabet  Aq  If  for  each  e > 0 there  exists  a sequence  of  binary 

variable-rate  codes  (called  an  e -weakly  universal  sequence)  such  that  for 
each  6 € A there  exists  an  Integer  N(0)  for  which  n ^ N(0)  Implies 

Pn(B.n.6)  ^ D + e 

and 

(ln2)"^[RQ(D)  + e]  . 

Loosely  speaking,  an  e-weakly  universal  sequence  Is  the  variable- 
rate  counterpart  of  the  weakly-minimax  universal  codes  of  [lO],  For  each 
0 € A,  weakly-mlnlmax  universal  codes  have  a fixed  rate  R and  an  average 
distortion  which  approaches  the  optimum  theoretically  attainable,  Dq(R), 

V 

whereas  an  e-weakly  universal  sequence  has  an  average  distortion  which 
approaches  D and  an  average  rate  which  approaches  the  minimum  theoretically 
attainable  rate,  Rg(D). 

Our  key  result  la  the  following  theorem. 

Theorem  2.  Suppose  la  an  abatract  alphabet  and  A^  Is  countable.  Suppose 
also  that  the  following  two  conditions  hold. 

3 b € Aq  3 Eg  p(XQ,b)  < CO,  V 0 € A (3) 


sup  {Rg(D)|9  € A}  < cc 


(4) 
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Then  the  class  [A/7,M'g|@  € A}  is  weakly  universal  encodable  at  distortion 
level  D with  reproduction  alphabet  A^. 

Remarks.  Notice  that  If  A^  Is  a metric  space,  (3)  Is  equivalent  to 

V e e A a bg  € Aq  9 Eg  p(Xg,bg)  < «.  (5) 

For  applications,  the  two  theorems  which  follow  are  considerably  more 
Important  than  Theorem  2;  both  theorems  use  the  following  condition. 

A 

Condition  (a'i.  There  Is  a countable  set  A^  c Aq  such  that  for  each  e > 0, 
each  9 € A,  and  each  finite  set  C c Ag,  there  exists  C'  c:  (Ag)  such  that 
Id  - Ic'l  and  EqP(x"1C)  s Egp(x"lo  + e.  In  [14],  Kieffer  shows  that  the 
following  two  Important  examples  satisfy  Condition  (a). 

Example  1.  Xg  Is  a separable  metric  space,  p Is  bounded,  and  p(x,»)  Is 
continuous  for  each  x € Ag. 

Example  2.  There  exists  a nondecreasing  continuous  function  f;[0,“)  -•  [0,») 
such  that  for  each  a > 0,  lim  [f (x4«)/f (x)]  ■ 1 and  there  exists  a metric  d 

X -.  00 

on  Ag  U Ag  such  that  Ag  is  separable  and  p(x,y)  f[d(x,y)]  for  each  x € Ag 

A 

and  y € Ag. 

Theorem  3.  Suppose  Ag  and  Ag  are  abstract  alphabets  and  f(A^,^g|6  € A}  Is 
a class  of  stationary  exgodlc  sources  such  that  (3),  (4),  and  Condition  (a) 
are  all  satisfied.  Then  the  class  {(A/T,pig)|0  € A}  is  weakly  universally 

A 

encodable  at  distortion  level  D with  reproduction  alphabet  Ag. 
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As  another  consequence  of  Theorem  2,  we  obtain  the  following 
result,  first  proved  by  Kleffer  [ 14] , which  is  the  most  general  almost' 
universal  coding  theorem.  It  includes  all  of  the  theorems  of  [l2]  and 
[13]  as  special  cases. 

Theorem  4.  Suppose  Aq  Is  an  abstract  alphabet,  (A,X,W)  is  a probability 
space  for  which  the  mapping  6 -•  M-0(E)  is  X-measurable  for  each  E € and 
Condition  (a)  is  satisfied.  Suppose  also  there  is  a A*  C X such  that 
W(A’)  ■ 1 and  such  that  the  following  two  conditions  are  satisfied; 

a b € Aq  3 Eq  p(XQ,b)  < »,  V e € A' 

and 

Rq(D)  < ®,  V 0 € A'. 

Then  the  class  {(Av^,pig)|6  € A}  is  W-alroost  universally  encodable  at  distor- 

A 

tlon  level  D with  reproduction  alphabet  A^. 

It  is  desirable  to  weaken  (4)  if  possible  since  there  are 
interesting  classes  of  sources  which  do  not  satisfy  this.  However,  we  note 
that  (4)  arises  naturally  in  many  important  situations. 

Example  3.  A^  finite  and  R0<D)  < * for  all  0 € A. 

Example  4.  A^  totally  bounded  under  metric  p and  R0(D)  < » for  all  0 € A. 

In  this  case,  for  each  e > 0 we  can  find  a finite  approximating  set 

A*  C Aq  for  which  Rg(D  -f  e , A*)  s *0(0)  for  all  0 € A,  where  Rg(D  + e.  A®) 

is  the  rate-distortion  function  for  reproducing  alphabet  A*  [12],  [l3]. 

Since  R0<D)  < ® for  all  0 € A,  it  follows  that  Rg(D  -f  «,  A*)  is  given  by 


I 
• I 

f 


6 I C I 

an  informaClon>theoretlc  minimization  and  therefore  R0(D  -f  e,  A ) ^ In  |A  | 

for  all  6 € A.  Thus  (4)  is  satisfied  and  applying  Theorem  3,  we  can  find  an 

£ 

e -weakly  universal  sequence  for  A and  A which  is  2g -weakly  universal  for  A 

A 

and  Aq. 

Example  5.  Compact  class  of  sources  and  R0(D)  < ® for  all  9 6 A.  If  the 

class  of  sources  is  compact  [17],  then  for  any  e > 0,  there  is  a totally 

bounded  set  A c A^  and  a mapping  t:  A "*  such  that  EgO(Xg,T  (X^))  < e for 

all  6 € A.  Under  this  condition,  it  is  easy  to  see  that  R0(D  + e,A®)  s Rq(D) 

£ 

for  all  9 € A,  Since  A is  totally  bounded  under  the  metric  p and  R0(D)  < 

for  all  9 € A,  we  can  apply  the  result  of  Example  4 and  obtain  an  e -weakly 

€ ^ 
universal  sequence  for  A and  A which  is  2e -weakly  universal  for  A and  A^. 

Example  6.  Class  of  covariance  stationary  processes  whose  variances  are 

A 

uniformly  bounded  with  Aq  ■ A^  ■ the  real  line  and  p the  squared  error 
distortion  measure,  [16,  pp.  134-136], 

Example  7,  Transmission  over  a fixed-rate  channel  of  capacity  C.  If  we 
are  to  reproduce  each  source  in  the  class  with  fidelity  D at  the  output  of 
the  channel,  the  converse  information  transmission  theorem  requires 
Rg(D)  £ C for  all  9 € A so  that  (4)  is  necessary. 

In  examples  3-7,  the  condition 


Rg(D)  < » V 9 € A 


appears  either  explicitly  or  implicitly.  This  condition  is  always  necessary 
for  A to  be  weakly  universally  encodable  since  otherwise  no  code  exists  which 
gives  average  distortion  D for  each  member  of  the  class.  To  replace  (4) 
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of  Theorem  2 by  the  considerably  weaker  requirement  (6),  we  require  that 
the  following  condition  hold. 

Condition  (b). 

(b.l)  For  each  0 € A there  exists  a real-valued  function  cpg  defined  on 
the  Integers  such  that  11m  cpg(n)  - 0 and  such  that  for  each  positive  Integer 

n -•  00 

n,  each  F € and  each  F € 7 

• -®  n n 

(b.2)  The  function  cpg  of  (b.l)  can  be  selected  such  that 

00 

2 [cp„(n)]^  < oo,  V 0 € A. 

n»l 

(b.3)  There  Is  a b € for  which 

Eg[p(Xo,b)]^  < ®,  V 0 € A. 

Remark.  The  following  Is  equivalent  to  (b.l). 

(b.l)'  For  eaca  0 € A there  exists  a real-valued  function  cpg  defined  on 

the  Integers  such  that  lim  cpg(n)  • 0 and  such  that  for  each  positive 

n — 00 

OD 

Integer  n and  each  E € 

with  probability  1 (note  that  the  conditional  probability  >Ag(E|?^)  ie  a 
random  variable).  The  equivalence  of  (b.l)  and  (b.l)'  Is  demonstrated  in 
[18].  If  p Is  a metric  on  Aq  U Aq,  (b.3)  Is  equivalent  to  the  following 


condition. 
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A r 2 

(b.3)'  For  each  9 € A there  exists  a such  that  EQ[p(XQ,bg)]  < ®. 

Examples  of  processes  satisfying  (b.l)  and  (b.2)  are  sequences  of 
Independent  Identically  distributed  random  variables,  sequences  of  m* 
dependent  random  variables  [19].  aperiodic  Markov  chains  satisfying 
Doeblln's  condition  [20]  . [2l]  and  stationary  Gaussian  processes  with 
rational  spectral  density  [2l]-[24],  Chains  of  Infinite  order  [25]  satisfy 
(b.l).  Note  for  example  that  the  class  of  zero  mean  stationary  Gaussian 
processes  with  finite  average  power  and  distortion  measure  p(x,y)  « |x-y| 
satisfies  (b.3). 

A 

Theorem  5.  Suppose  Is  an  abstract  alphabet  and  A^  Is  countable.  If 
Condition  (b)  Is  satisfied  and  if  ^ “ for  each  0 € A,  then  the 

class  Is  weakly  universally  encodable  at  distortion  level  D with  reproduction 
alphabet 

A 

Corollary  1.  If  in  Theorem  5,  A^  is  given  by  either  Example  1 or  2,  then 
the  conclusion  Is  still  true. 

Finally  we  mention  that  A countable  Is  also  a sufficient  condition 
to  replace  (4)  of  Theorem  2 by  (6).  This  can  be  shown  using  techniques  of 
the  proof  of  Theorem  2. 
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Chapter  4.  Strongly  Universal  Variable  Rate  Codes 

A 

In  this  chapter,  we  assume  for  convenience  that  and  that 

the  distortion  measure  p satisfies  Inf  {p(x,y)|y  € Aq}  » 0 for  all  x € Aq."*" 
Definition  2.  A family  {(A/7,M>-|9  € A}  of  stationary  sources  Is  strongly 
universally  encodable  at  distortion  level  D If  for  each  e > 0 there  Is  an 
Integer  n^  and  a binary  variable  rate  code  on  blocks  of  length  n , B , such 
that  for  each  9 € A, 

Pn  (B^  ,e)  s D + e 
e e V 

and 

?n  .9)  ^ (ln2)'^[Rg(D)  + 0]. 
e e 

If  strongly  universal  codes  exist  for  distortion  D for  a family  of 
sources  with  Index  set  A,  then  there  Is  a single  binary  variable  rate  code 
which  has  average  distortion  approximately  D and  average  rate  approximately 
R0(D)  for  each  9 € A.  Such  codes  are  the  variable-rate  counterparts  to 
strongly  mlnlmax  universal  fixed-rate  codes  [8] . 

Naturally,  strongly  universal  codes  do  not  exist  for  as  broad  a 
class  of  sources  as  weakly  universal  codes,  and  later  In  the  section  we  give 
results  that  substantiate  this.  First,  however,  we  give  two  sufficient 
conditions  for  strongly-unlversal  codes  to  exist,  the  first  Involving  the  p 
distance  measure  between  random  processes  [lO]. 


Theorem  6.  Suppose  Aq  Is  a finite  alphabet  with  metric  distortion  measure  p. 
Then  any  family  of  stationary  ergodlc  sources  [(A/7,|ig)|e  6 A]  totally  bounded 
under  the  p distance  la  strongly  universally  encodable  for  any  nonnegative 
distortion  level. 


I 

I 


I 

I 
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We  now  give  another  sufficient  condition  for  the  existence  of 
strongly-unlversal  codes.  Following  [26],  we  say  a binary  variable  rate  code 
fB  ,U  ,V  ) is  a D-bounded  distortion  code  if  p(x'^,V  [u  (x”)]  ) ^ D for  all 
x''  € Aq.  For  integers  n s 1,  0 € A,  and  D s 0,  define  the  OPTA  function 

r^(n,D,9)  = lnf{ r^(B^  , 0)  |b^  D-bounded] 

This  is  well  defined  since  B([Aq})  is  a D-bounded  distortion  code 
for  all  integers  n ^ 1,  0 G A,  and  D ^ 0.  Define 

r^(D,0)  * inffr^’CnjD, 0) I n s l] . Standard  subadditivity  arguments  can  be  used 

to  show  r^(D,0)  * 11m  r^(n,D,0). 
n 

For  any  Integer  n ^ 1,  we  can  form  a binary  variable  rate  code  by 

combining  the  binary  bit  representation  for  a block  code  provided  by 

^ n 

Theorem  1 with  the  binary  bit  representation  for  A^,  using  the  encoding  rate 
of  mapping  a source  word  x into  the  shortest  bit  representation  associated 
with  a reproducing  word  for  which  p(x,^)  ^ D.  In  this  way,  we  can  prove 
Proposition  1. 

Proposition  1.  r^(D,0)  • D > 0,  (A,i7,|ig)  a stationary  ergodic  source, 

and  |Aq1  < ®. 

Proposition  2.  Suppose  Aq  Is  finite  and  D > 0.  If  r^(n,D,0)  converges 
uniformly  on  A,  then  any  family  of  stationary  sources  {(A/7,)Xg)|0  € A}  for 
which  r*^(D,0)  • 0 € A is  strongly  universally  encodable  for 

distortion  level  D. 

The  condition  r^’cD.O)  ■ Rq(D),  0 € A,  is  automatically  fulfilled  if 
{ (A,i:7,^g)  1 6 € a}  is  a class  of  stationary  ergodic  sources.  As  an  application 
of  Proposition  2,  we  have 


1 

i 

\ 
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Theorem  7.  The  class  { (A/7,(0,q)  1 6 € A}  of  all  binary  first-order  Markov 
sources  is  strongly  universally  encodable  for  any  nonnegative  distortion 
level . 

The  following  result  prepares  us  to  give  partial  converses  to 
Theorems  6 and  7.  Following  this,  we  give  two  propositions  which  show  that 
strongly-unlversal  codes  do  not  exist  for  a class  of  sources  without  restric- 
tions on  the  memory  and  source  alphabet  of  the  class.  The  proof  of  Proposition 
4 uses  a construction  contained  in  [S] . 

Proposition  3.  Suppose  Aq  is  finite.  If  a family  of  stationary  ergodlc 

sources  {(A,i7,Hg)|0  € A}  is  strongly  universally  encodable  for  positive 

distortion  level  D,  then  for  any  e > 0,  there  is  a positive  integer  n depending 

on  e such  that  R q(D)  < Rq(D)  + e for  all  0 € A. 
n,  o o 

Proposition  4.  Suppose  Aq  Is  finite  with  metric  distortion  measure  p.  The 
class  of  all  finite  order  Markov  sources  { (A,C7,p.q)  | 9 € A}  is  not  strongly 
universally  encodable  for  any  distortion  level  D satisfying 

0 < D < J ^min{  E p(x,y)|x  € A }. 
ye  Ao 

Proposition  5.  Suppose  A^  is  a metric  space  with  unbounded  metric  distortion 
measure  p.  The  class  of  all  memoryless  sources  { (A.f^.M'g)  | 9 9 A}  Is  not 
strongly  universally  encodable  for  any  nonnegative  distortion  level  D. 
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For  each  k€  K,  define  a function  c^^:  Aq^->  C-(k)  as  follows: 
for 

(1)  If  (*.(k,x  0,  let  Cj^(x  ^ be  the  code  of  least  Index  in 

(il)  If  * 0»  let  Cj^(x*^^  be  the  code  of  least  Index 

In  C(k)  for  which 

I IS  <1<  v,^). 

Note  that  Cj^  is  well  defined  since  C-Ck)  is  non  empty  for  each  k. 

Claim  1.  For  k€K.  Cj^  is  - measurable. 

Proof.  For  C(k,p)  €C-(k) , 1 $ P < we  have 

lx"‘«K(x"n-C(k,p))  . tx"“‘>  |a(k.i”«)  «,  c^Cx"^"')  . C(k,p)] 

U(x"<">|0<k.x"«)  . »,  c^(x"<">)  . C(k.p» 

• ({fj^^l  > D+e/2}  n [fj^^2  >D  + c/2]n...n{fj^  p_j  > D+c/2} 

n(f.  „ < D+€/2])U({f  >D  + e/2}n...n{f,  >D  + e/2} 

X»  P •kf  J-  ’'^k 

n{fk^p  - minCf^^J  1^  *1  ^ ^ ^ 


.1 


.! 


n(k) 


For  each  k?  K,  define  hj^  : A”*  [0^®)  by 


1 

I 

I 
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S p[x‘‘(T%)|c  (X"^‘"^U)))] 
1-0  ^ 


Claim  2.  For  k € K,  hj^  Is  C7-roeasurable. 
Proof.  For  or  a real  number 


fu)(h^((u)<Qr]  - {<“|cj^(x"^*"\tu))  - C(k,q),fj^  q(x"^‘"^(ua))  < o)€  O. 


iSq^v, 


For  each  k C K and  each  Integer  1,  1 s i s v , let  w be  the 

k ^ 

(log2k)  bit  binary  representation  of  1,  and  let  (B^,  U^,  V^)  - B(C(k,l)). 

K K Ic  * 


For  e.ch  k f k.  define  the  code  by 


(b) 


{x6  aJ^*‘^|cj^(x)  - C(k,l)},l-l,...,v  . 


,n(k) 


k 

^ '■®k^  *'.l-l,...,Vj^e 

Clalm_2.  For  k € K,  Is  -measurable. 

froaf.  - (‘k.i.VXl ‘l  .1»  ■ - C(k. 


D) 


n( 


,n(k) 


0 < 


n u)  - b ))€ 

f ^ ' k"~  'r,k'  “O 

For  each  k € K,  since  *®b  of  binary  sequences  satisfying  the 

yr.fi>  eoMltlen,  1.  tf;<'‘>-«..Mr.bl.,  end  *n(k)!»n(^)  - 


r 
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) Is  a binary  variable  rate  code  on  blocks  of  length  n(k) 


Note  that  from  (1),  for  each  kfK  and  each  A 


Average  Distortion  of  Codes 


Assume  for  the  moment  the  truth  of  the  following  lemma 


Lemma  1 . For  each  there  is  an  Integer  r(fl)  <:  K and  a code 


C(9)6i3  (r(9))  containing  b 


for  each  positive  integer  J and 


< (In  2)  [Rg(D)  +(e/2)] 


In  the  remainder  of  the  proof,  we  fix  0€A.  For  r(9)  given 


by  Uma  1,  and  integers  k>  r(6),  k€K,  define  the  random  variable 


1-0 


r 
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Note  that 


r(0)’^n (k)-l 

SM  • r(e)’^n(k)  E |c(9)I 

* i-0 


and  therefore  from  Lenina  1, 


^iJg.  > D + (c/2)}  < 6*(9) 


For  k > r(9),  k€R,  from  (14) 


log2l[C(0)}>^/’'<®)  I - 1082!  C(6) 

< r(0)'Hc{r(9)(ln2)"^  rRg(D)  + (c/2)J  + l} 

< k((ln2)"^Rg(D)  + 2) 


It  followa  that  for  all  large  k,  k€K, 


k’^r  log2|[C(0)]'‘/’'<®>  |1<M.  (17) 

Now  chooae  k(0)  > r(0),  k(0)€K,  auch  that  k > k(0),  k^K  Impllea  inequality 
(17)  la  aatlafled  and  C(0)c  [AQ(log2k)]’^^®\  Then  for  k > k(0),  k€K, 


[0(0)}*^^'^®^  c [Aq  (log2k)J*' 


and  alnce 


For  k>  k(0),  k€K,  (17)  through  (19)  Imply 

rc(e)j‘‘^'^*^  €C.(k) 
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We  now  establish  three  claims. 

Clalm_4.  For  k€K,  h^(<D)  > D + (€/2)  implies  C^(k.  ). 

Proof-  Tf  3(k,  jt  0,  then  by  (i),  hj^(uj)  £ D + (e/2),  a contradiction. 

Claim  5.  For  k > k(9),  k^K,  hj^(!^)  > D + (e/2)  Implies  gj^(u))>  D (e/2). 

Proof.  By  Claim  4,  ^(k,  }f ^*^\u)))  - 0.  Since  [c(e)]*''^’^^®^€  C-(k)  by  (20), 

(8)  implies  gj^(uj)  > D + (e/2). 

Claiis_6.  For  k > k(9),  k€K,  h^^  < on  {h^  > D + (e/2)} 

Proof.  By  Claim  4.  h^(o.)  > D + (c/2)  implies  CJ(k,  x"^*^>(u;))  - 0.  Since 
rc(9)]  €C-(k),  (it)  implies  h (ou)  < g,  (uj). 

K ”*  iC 

Note  that  Claims  5 and  6 imply 

^9  k ^ {hj^>  D + (e/2)]  - ^9  ®k  ^{gj^>  D -F  (c/2)] 

for  k > k(9),  k€K,  where  Xp  is  the  indicator  function  for  the  set  F,  F € c7. 

Finally,  using  (9)  and  (21),  we  have  for  B^  for  k s k(9),  k € K, 


o(k), 


^9^^  D + (c/2)]  '^^[hj^>  D+(e/2)]  ^ 

Note  that  alnce  b'<®>  €0(9).  g^  < p(x‘<»'),b "<*'>)  .„d  thus 

*k^{gj^>  D+  (e/2)}  - > D + (e/2)} 

i P(9)»*9fg,j  >D-F(e/2)}+  EQP()^'<*‘>,b"^‘‘>)  x 

‘ ~ • fp(S"^''\b*^‘'^)>p(9)} 

(23) 
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Call  the  eecond  term  on  the  right  aide  of  the  above  inequality  q(n(k),8). 
Combining  <U).  (IS).  (22),  and  (23)  givea 


Pn(fc)(9)  < D + (3«/4)  + q(n(k),0)  (24) 

for  k > k(9),  k€K.  Under  the  conditlona  of  the  theorem,  Berger  [16]  haa 
ahown  q(n(k),6)  ^ 0,  k € K. 

Average  Rate  of  Codea 

For  k>  k(9),  k€K,  and  <*)  € [g^ < D + (e/2) } , (8)  and  (20)  imply 

[C(0)]*^/'^®V  ^(k.x"^*^^^)). 

Thua  from  (1), 

k ^f'log2lcj^()f^*'^U)))|')  s k"^riQg2)[c(0)]'^^’^^®^l] 

< r(0)"^riog2lc(0)n  + k“^  . (25) 

Uaing  (10)  and  (25)  givea 

^ '<®>'^'^(‘^)rio82|c(9)l]  + k"^n(k)+(log2k)''  (26) 

Then  for  k > k(0),  k€K,  uaing  (7),  (10),  and  (26),  we  have  for  g 

-n(k)’ 

'n(k)^*^  " «0«(k)  ^*‘^]0Cfg^sD+(t/2))  ■^’^{g^^>D+(e/2)}^ 

< r(0)"^riog2lc(0)|^  + k’^  +MHg{g^  > D + (c/2)}+n(k)’^(log2k)‘‘ 

(27) 

Uaing  (13)  through  (15)  givea 

?^lj)(«)  < (ln2)"^[»g(D)  (3«/4)]  + k*^  + n(k)‘^(log2k)*‘  (28) 

for  k > k(6),  k€K«  The  laat  two  terma  decreaae  to  0 with  k,  k€K. 
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Construction  of  Sequence  of  Codes 

From  this  sequence  of  codes  ^ we  can  construct  a sequence 

of  binary  variable  rate  codes  on  blocks  of  lengths  1,  2,  3,,..  which  will 
have  the  asymptotic  properties  of  the  original  codes.  It  suffices  to 
show  how  to  construct  a binary  variable  rate  code  on  blocks  of  length  n for 
an  Integer  n satisfying  n(k)  s n s n(k+l).  We  follow  the  construction  of  [12, 
Appendix  A]  first  using  many  times  as  possible  on  the  source  word  x^" 

of  length  n,  then  using  B as  many  times  as  possible  on  the  remaining  n 

mod  n(k)  letters,  and  continuing  in  this  fashion  until  the  number  m of  remain- 
ing letters  is  less  than  n(l).  As  In  [12] , this  defines  a binary  variable  rate 
code  on  blocks  of  length  n,  B^,  where  the  encoding  mapping  ignores  the  m 
remaining  letters  and  the  decoding  mapping  adds  a sequence  of  m b's,  to 
give  a full  n-tuple  of  reproducing  letters. 


If  we  let  a^^^(ii)  = n,  aQ(n)  = 0,  and  a^.  (n)  = n(i) 

for  1 £ 1 ^ we  see  that  " Sj(n)  is  the  number  of  letters  of 

X encoded  using  B ...  for  1 ^ 1 s k,  and  It  Is  the  number  m of  letters 

Ignored  by  U_  for  1*0.  The  average  distortion  of  B is 
n 


p^(0)  - n'^{^Z^fa^^j(n)  - a^(n)lp^^^^(0)+[aj(n)  ‘a^CnljEg  pCXg.b)}  (29) 
and  the  average  rate  la 

;^(0)  - n-i  OO) 

It  follows  from  (24)  and  (28)  that  there  Is  an  Integer  m(9) 6 K such  that 
for  m > m(0),  mg  K, 


TTi(f^)  - 1,  from  (7)  and  the  code  construction 


Then  for  n > n(m(0))  we  have  from  (29)  and  (30)  that 


P_(6)<  n [n(D  + (7e/8)) +n(m(e))E.p(X.,b)] 


rn(9)<n  [n(ln2)  [Rq(D)  + (7€/8)l +n(m(e>)  max{r  ^ i ^ m(0)-l}] 


It  follows  that  there  is  an  Integer  N(P)  such  that  for  n > N(P) 


It  only  remains  to  prove  Lenma  1 since  (32)  and  (33)  establish  the  theorem 


Proof  of  Lemma  1.  In  this  proof,  we  fix  9 ^ A.  All  chosen  constants  and 


defined  functions  depend  on  the  particular  9€A  fixed,  but  for  convenience 


this  Is  not  explicitly  denoted.  Choose  s > 1 such  that 


Choose  6 > 0 small  enough  such  that  ( < 1 and 
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Choose  an  integer  s > 0 such  that  for  Integers  t > a, 

(D+a6)"S(^)q(t,9)<  8"^  c (37) 

(D +a6)'^Mq(t,0)  < (8  ln2)'^  e (38) 

Since  Ep  p(XQ,b)  < * and  R(^(D)  *»  we  can  apply  Theorem  1 to 

obtain  for  an  integer  r > a,  r € K,  a code  (r)  containing  b and 

satisfying  (14)  and 

^i9{p(X'|c)  > D + 6}  < 8 (39) 

We  now  show  that  we  can  find  a constant  8*  which  satisfies  (11)  through  (13). 

Define  S€(7by  S - {p  (x'^|c)  €(D  + 6 , D +a8)}  , and  define  T€<7 
by  T = ( D (x'^lc)  > P (0)}  • Define  random  variables  p:  A"*  [0,“)  and 
q : A [0,®)  by 

P - (D+8)x^  + P(X*^|C)X  ^ 

q < P < P(x’^|c)  < p+a6  (40) 

P - q + P(x’"|c)Xt  (^1) 

Then  for  each  positive  integer  Jt  using  (36),  (40),  and  (41), 

P(X*^  •T^'|C)>  D+ («/2)}  ;S  p(X*'«  T^'^|c)>  D + 2a6} 

® i-0  i-0 

< V(p.T^")>  D+a6) 

® i-0 

• P(X'*  C)(X^»T^')1>  D+a«) 


and 

Note  that 
and 
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(q»T^*^)>  D + a6}  + P(x"^  » T^’'|c)(  x ^ * T^'')  > D +«M 
i«0  1-0  ^ 

(42) 

where  the  last  Inequality  follows  from  the  union  bound.  Let  and  I2 
denote  the  first  and  second  terms  given  by  the  union  bound.  From  the 
Chebychev  Inequality, 


I,  < (D+a6)‘^E.  "i:  p(x’'<»T^''|c)(x-'T^’') 


r.  „lri 


- (D+afi)"^EgP(x’'|c)x^ 

S (D+a6)‘^E.  PC/.b’^lx  ^ ^ 

® {o(X^b’^)>p(e)] 

= (D  +a6)  ^ q (r,0) 


We  now  consider  Ij.  Note  that 


J-1  j-1 


)-i  i,  j-i  j,  .1-1 


(45) 


For  u)  such  that 


(45)  gives 


E q(T  ou)  > J(D  + a6), 
i-0 


(D  + 5)Vx,  ^ ^ + p(6)^A(  ^ 0 + > J(I>  + *5). 

t«o  ^ ^ * 1*0 

and  further  algebra  using  (44)  reduces  this  to 

Ir  -1 

J a D + afi)^’^  ^ (a-l)6. 
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Proof  of  Theorem  3. 

Let  R'(*)  be  the  rate-distortion  function  for  source  (A,i7,u.-)  and  reproducing 
9 U 

alphabet  A',  0 6 A.  We  have  from  Condition  (a)  that  Rl(D  + e)  ^ Rq(D)  for  each 
U 9 9 

€ > 0.  Since  Rq(*)  is  a continuous  function,  this  implies 

R^(D)  5 Rg(D)  . (48) 

A 

Condition  (a)  also  Implies  that  for  b <z  A^,  there  is  a b'  G A^  for  which 

Eg  p(XQ.b')  £ Eg  D(Xg,b)  + I < cc  (49) 

Equations  (48)  and  (49)  enable  us  to  apply  Theorem  2 to  obtain  for  e > 0, 
a sequence  of  codes  [b^}  and  for  each  9 G A an  integer  N(0)  such  that 
for  n 2 N(9), 

p (0)  s D + e 

and 

r^  (0)  i (ln2)’^[R^(D)  + e] . 

Applying  (48)  gives  the  result  of  the  theorem. 

Proof  of  Theorem  4.  Let  e > 0.  As  discussed  in  [14] , we  can  assume  in  proving 

the  theorem  that  Rg(D)  < “ for  all  0 G A.  This  condition  and  Condition  (a) 

imply  that  the  map  0 - Rg(D)  is  T-measurable  [14].  Since  the  sets 

A »f9GA|R„(D)^n}GT  Increase  to  A , we  can  find  a set  A„  G £ with 
n ' o N 

W(A^)  2:  I - (e/2)  for  which  0 G A^^  implies 

sup  [Rg(D)l0  G Ajj}  £ N (50) 

and 

Eg  p (Xg.b)  < » (51) 

Using  (50)  and  (51)  to  apply  Theorem  3,  we  obtain  a sequence  of  codes 
and  for  each  6 G A^  an  Integer  N'(6)  for  which  n N'(0)  implies 

0^(9)  s D + (e/2) 


and 


?^(9)  S (ln2)“^[RQ(D)  + (e/2)]. 

For  each  n,  define  functions  f :A„  -•  [0,®)  and  g :A„  -•  [0,®)  by 

n N n N 

fn(®)  ■ supfoj^(9)|k  s n] 

and 

g^(0)  - sup{rj^(9)|k  2r  n] . 

We  have  that  f^  and  g^^  are  {b  H Ajj]B  € l} -measurable  functions  and 

11m  f (9)  s D + (e/2), 
n 
n 

11m  g (9)  (ln2)"HRfl(D)  + (e/2)], 
n " ® 

for  each  9 € Aj^.  From  Egoroff's  theorem,  we  can  obtain  a set  A'  € X with 

W(A')  s 1 - e and  an  Integer  N*  such  that  for  9 € A'  and  n 2 N*, 

f (9)  s D + e 
n 

and 

g^(9)  (ln2)’HRg(D)  + e]. 

This  establishes  the  theorem. 

Proof  of  Theorem  5.  With  the  exception  of  an  analogue  to  Lemma  1,  the 
proof  of  Theorem  5 Is  very  similar  to  the  proof  of  Theorem  2.  Therefore, 
we  only  Indicate  necessary  changes  to  the  proof  of  Theorem  2. 

First  let  D ^ 0 satisfy  (D)  < a»  for  all  0 6 A*  I”  place  of  (7), 
we  make  the  definition 

C-(k)  - {C  eC*(k)|C  c [Apdog^k)!*',  b**  € C,  and  fcj  ^ J^]  (52) 

In  place  of  Lemma  1,  we  have 

Lessna  2.  For  each  0 € A>  there  is  an  integer  r(0>  € K and  a code  C(0) 
containing  Ji.  and  satisfying 

p<x'^®^|C(&))  > D + (e/2)}  < (J  eS'^  16a^ 

^ l-O  ® 

for  each  positive  Integer  J where  a^  ^ * *"^<1 

9 


(53) 
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r(e)”^  riog2lc(e)p  < (ln2)  ^ [l^  (D)  + (e/2)]. 


(34) 


Using  (53),  we  have  corresponding  to  (15) 

^ ^ 16(nj^e^)”^r(6)ag. 

From  (16),  we  have  for  all  large  k ^K, 

Xog2l[C(e)l*"^''^®^  I < k log2  k - log  k, 


(55) 


Implying 


lrc(0)]‘'^’^^®^  I < k"^  2^  ^ 


which  replaces  (17).  As  in  the  proof  of  Theorem  2,  we  can  then  show  there 
is  an  Integer  k(0),  k(0)  € K,  for  which  k > k(©),  k € K,  implies 
rc(0)l*®^’^^®^  €.3(k).  Then  from  (22),  (23),  and  (55),  we  have 


2.-1 


^n(k)^®^  ^ ^ (e/2)  + 16(nj^e  ) p(0)r(0)a  + q(n(k),9) 


(56) 


for  k>k(0),  k^K. 

We  now  examine  the  average  rate  of  F«>ni  (10),  (26),  and 


(52), 


*'n(k)  (fi)  < '^(&)’"riog2|C(0)n  + k"^  + k"^flog2 

+ n(k)’\log2k)’" 

for  k > k(0),  k € K.  corresponding  to  (27).  Substituting  (54)  and  (55) 

in  the  above  equation  gives 

,-l 


^n(k)®^  “ (ln2)"^[R^(D)  + (e/2)]  + 16(ek)‘2{ Jj^“ Yl°«2JkP'^®>*0 
+ k”^  -»■  n(k)“^(log,k)‘‘ 


(57) 


for  k ^ k(0),  k € K.  The  last  three  terms  decrease  to  0 with  k,  k g K 


(J,,  -*  •). 

k 


■JTT- 


To  construct  a sequence  of  binary  variable  rate  codes  on  blocks 


of  length  1,2,3,...,  we  follow  the  argument  of  Theorem  2,  using  (56)  and 
(57)  instead  of  (24)  and  (28)  and  using  (10)  and  (52)  to  replace  (31)  with 

^n(i)<®^  ^ 2"^riog2J^l  + n(i)‘\log2i)^ 

for  i*l, . . . ,m(9)-l.  It  only  remains  to  prove  Lemma  2,  but  first  we  prove 
a useful  lemma. 

Lemma  3.  Fix  0 € A.  For  q a positive  integer,  let  g be  an  A^-measurable 
function  such  that  for  f ■ g • X*^,  2gf“0  and  Eg|f|^  < ®.  Then  if  (b.l)  and 
(b.2)  of  Condition  (b)  hold. 

In  2 

E [ E f - « a J 

® i-0  ® 

for  each  positive  integer  J where  a^  < <». 

Proof  of  Lemma  3.  Let  J be  a positive  integer. 

Eq[  ^E  f o E J^E  [ f o T^*^  2E  (f  c t"“*)  (f  o t“**)1 

1-0  *'1-0  0 s m < n s j-1 

S ^E  E [foT^*’]^+  2E  |E  (f  o T"^)(f  o t"**) 

i-0  ® 0 s m < n « j-1  ® 

« J Eq  f^  + 2J^E  |e  [f][fc  t’'*']] 
k-1 

s jfB  f*  + 2 E |Eg[f][f  e T*“*]  I) 
k-1  ® 

Under  the  hypothesis  of  the  leans,  It  follows  from  [18,  Leona  1.1]  that 
|EQ[f][foT^‘*]|  < 2[<pg(kq)]\|f|* 
for  k-1, 2,,.,  Then  itslnf  (58), 

E^cV  < j{Ea  f*  * S [<Pa(kq)]'} 

® 1-0  8 B 9 


(58) 


establishing  the  li 


Some  chosen  constants  and  defined  func 


,emma 


tions  depend  on  the  particular  0 € A fixed,  but  for  convenience,  this  is 


not  explicitly  denoted.  Since  by  hypothesis,  o (X-,b)  <; 


we  can  apply  Theorem  1 to  obtain  an  integer  r € K and  a code 


C € C^*(r)  containing^  and  satisfying  (54)  and 


Define  a function  f:  A -•  To,® 


Then 


Since  CondiClon  (b)  holds,  (61)  gives 


Using  (60)  and  (62)  to  apply  Lemma  3 gives 


for  each  positive  Integer  j where  a 
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For  each  positive  Integer  j 
P (X^T^’^lO  > D + (€/2)1 

^ i-0 

-1  r ir 

- Pr{J  Z [p(X  *T^  |c)  - (D  + 6)1  > ^(e/2)  - 6]1 

^ 1=0 

ir 

- Ifc  { Z > jr(e/2)  - 6]1 

^ i-0 


Applying  Chebychev's  ineouallty  to  the  right  hand  term  and  using  (63), 
we  have  for  each  positive  Integer  J, 


-1  r Ir  o 4 

lfc{J  r (X^t"|c)  > D + (*/2)1  ^ (J[(e/2)  - 6])*'2e:[  j; 

1-0  ^ i-0 


lr,i2 


^ (J(€/4))"2  a^J 


where  a < ®,  which  establishes  (53). 

9 

Proof  of  Corollary  1.  In  the  case  of  Example  1 or  2,  Kleffer  shows  that  the 
Aq  of  Condition  (a)  can  be  taken  as  a countable  dense  subset  of  As  in 

the  proof  of  Theorem  3,  we  can  show  that 


(D)  < (D) 

for  all  0 € A where  R^(*)  Is  the  rate-distortion  function  for  source 
(A/7ti^)  reproducing  alphabet  A^.  Under  the  hypothesis  of  Corollary 
1,  this  Implies  1^' (D)  < ® for  all  0 € A*  To  apply  Theorems  to  the 
family  of  sources  ((A^CFip^ ) |9  € a1  with  reproducing  alphabet  A^.  we  must 
In  addition  show  that  there  Is  a b*  € for  which 

^rp(XQ.b')]^  < •,  e € A* 


(64) 

(65) 


(66) 


(67) 


We  show  (67)  in  the  case  of  Example  1,  Since  o is  bounded  and 

2 2 

o(x,*)  is  continuous  for  each  x € A^,  p is  bounded  and  (o(x,*))  is  con- 
tinuous for  each  x € Aq.  From  Condition  (b),  there  is  a b € A^  for  which 
2 

Eg[p(Xo,b)]  < ®,  and  from  Condition  (a),  there  is  a b'  € A^  for  which 
E0[o(Xotb' )] ^ < EgCoCX^b)]^  + 1 < »,  establishing  (67). 

The  proof  of  (67)  in  the  case  of  Example  2 is  similar  in  spirit. 
We  can  now  apply  Theorem  5 to  obtain  for  e > 0,  a sequence  of 
codes  [b^}  and  for  each  0 € A an  integer  N(9)  such  that  for  n a N(0), 


and 


P„(9)  « D + e 

?^(0)  ^ (ln2)"^[R^(D)  + e]. 


Applying  (66)  gives  the  result  of  the  theorem 
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Appendix  B 

Proof  of  Theorem  6.  Let  D ^ 0 and  € > 0.  Let  p be  a maximum  value  of  dis- 
tortion. First  choose  a > 1 large  enough  and  then  6 > 0 small  enough  such 


that 


a6  + p^  ((a-1)  ^2  + 6 ) < e 

(68) 

(InJ  + 6)((a-l)"^2  + 6)  < e/4 

(69) 

and 

6 < e/41n2 

(70) 

Since  {^0|d  ^ a}  la  totally  bounded  under  p,  A can  be  partitioned  Into  a finite 
collection  of  sets  k ■ 1,...  K with  the  property  that  0,  0*  € A^  Implies 
p(0,  0*)  < 6.  For  k • 1,...  K and  Integers  A - 1,.,.  f^lnjl  , define  sets  \ ^ \ 

\,i  " ^ ^ Ujt-i)fi.  A6)3. 

We  relndex  the  family  of  sets  l^k^K,  l^A^  TAlnjl  to  obtain  a 

family  of  sets  A , m ■ 1,...  M.  For  m - 1,...  M,  label  a parameter  In  A by 
HI  in 

0 ; It  follows  that  for  0 € A , we  have 
m m’ 


p(0,9J  < 6 


(71) 


and 


|Rfl(D)  - Rft  (D)|  <6 


(72) 


m 


Theorem  1 implies  that  for  the  finite  family  of  sources  |m  - 1,...  m} 

m 

there  is  an  integer  n and  block  codes  of  length  n C^(*)»  ■»  * !*•..  M, 
setisfying 


end 


>*0  CpQl"  I c„(«))  > D + 6)  < 6 
n’^riotjlv^^n  < a«2)'^[Re  (D)  •*■6] 


(73) 

(74) 


D 


M,  We  Incidentally  assume  n Is  large  enough  to  satisfy 


We  now  use  methods  of  Cl2]  to  combine  the  M block  codes  Into  a binary  variable 
rate  code.  It  follows  from  (74)  that  each  code  book  C (m)  has  a binary 


representation  B^(m)  of  |c^(m)|  binary  words  of  length  less  than 

CRq  (D)  + 6]  bits.  Denote  the  correspondence  between  codewords  of 
m 

B (m)  and  C (m)  by  the  bijective  mapping  V :B  (m)  - C (m).  Prefix  each 
“ n n,m  n n 

codeword  In  B^(m)  with  the  Tlog^Ml  bit  binary  representation  of  the  Integer 
m,  and  denote  the  resulting  codebook  by  6 (m).  The  mapping  ^ ;B  (m)  -•  C (m) 

n iijin  n n 

A 

Is  the  bijective  mapping  for  which  ^(c)  is  just  the  prefixed  version  of 

for  £ € C (m). 
n^in  ••  n 

Let  B^  - U B (m)  and  define  the  mapping  V :B  -•  A*  by 
n_n  *^  nnO'^ 


B Is  defined  as  follows 


If  min^p(x  |c^(m))  s , let  U^(x”)  be  a minimal  length  codeword  b € B^ 
which  satisfies  p(x”,  V (b))  s D + a6 . 


(Bn.Un.Vn)  forms  a binary  variable  rate  code  on  blocks  of  length  n 


In  the  remainder  of  the  proof,  we  fix  0 € A.  It  follows  that  0 C A for 


soma  Integer  m.  It  follows  from  the  coda  construction  that  the  average 


dlatortlon  satiaflea 
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and  using  (72),  Ch«  average  rate  satisfies 

?^(6)  s (ln2)"^[RQ(D)-»«]  (ln2)“^(lnJ+6)Hg{p(x”|c^<m))  > D-f«6}  + n'^fiog^Ml  (77) 


To  proceed  further,  we  need  the  following  lemma,  whose  proof  we  defer. 


Lemma  5.  p(x"|c^(m))  > D -t-  a&}  < (a-1)’^  2+6. 

Using  (78)  in  (76)  gives 

P„(0)  s D + a6  + pj(a-l)"4  +6) 

and  by  (68) 


P„(0)  < 0 + e. 

Using  (78)  in  (77)  gives 

f^(0)  « (ln2)’^[Rg(D)  + ^ + (ln2)*^(lnj  + 6 )((a-l)’^  + 6 ) + n"^  ^log2^^^ 

and  by  (69),  (70),  and  (75) 


(78) 


(79) 


r„(0)  < (ln2)’^[Rg(D)  + e]  (80) 

Since  (79)  and  (80)  hold  for  9 € A and  n 2 n*,  the  theorem  will 
be  established  with  the  following. 

Proof  of  Lemma  5.  From  the  theory  of  the  p distance,  there  is  a Joint  measure 
p**  on  A**  X A°  having  and  as  marginals  for  which 

0 n 

x")p"(<*£".  <*i“)  ^ 

^ aJ  “ ■ 

« P(0,0.)  6 

<2  6.  (81) 

where  the  last  Inequality  follows  from  (71). 


Define  the  set  S c a”  and  the  set  T c a"  by 

0 0 

S ■ f2L"|p(2.”|C^("'))  > D + a6} 

T *=  ^ D + 6} 

for  (x^.x!*)  € S X T,  p(x",x")  > (a-l)6.  Thus  from  (81) 

2 * > J P(2.'*,i")p”(dx",di") 

S X T 

> (a-l)fi  p"{S  X t} 

From  (73)  and  (82), 

- p"[s  X aJ} 

- p"[s  X t}  + p"{s  XT*^) 

< (a-l)’^2+  p"{aJ  X T*"} 

- (a-l)"^2+  nj  {t‘=} 

m 

< (a-l)‘^2+  6. 

Proof  of  Proposition  2. 

This  proof  Is  the  variable  rate  analogue  to  [8,  Theorem  5.1].  For  e > 0, 
choose  n large  enough  such  that 


r*’(n,D,0)  s Rg(D)  + s/2  (83) 

for  all  OCA.  The  above  bound  can  be  achieved  for  all  0 € A by  considering 
codes  from  the  smaller  class  C of  D-bounded  distortion  codes  B for  which 

< |Aq|,  2“  € Aq,  since  for  any  code  B'  ^ C,  there  Is  a code  B"  € C 
of  the  same  average  distortion  and  no  greater  average  rate  gotten  by  changing 
B*  Into  a RuffsMn  code.  Since  |c|  < •,  we  may  choose  k sufficiently  large  that 


42 


(nk)"^  flog2|cn  < e/2  ( 

and  index  the  codes  In  C as  1«1 |c|.  For  let  be 

the  riog2|c|1  bit  binary  representation  of  i,  and  form  the  cancatenated 

binary  variable  rate  codes  consisting  of  binary  codebook 
1 Ik 

B^l^  « X [B  ] and  the  natural  encoding  and  decoding  mappings 

- (U^,U^ U*')  and  - (V^,V^ V^).  Combine  the  codes  to  form  a 

code  B , as  follows; 

-nk 

(a)  B , - U , 

**  1 « 1 s:  Icl 

(b)  UnkCs”*")  - 

^Cui.  (x”*")]  - min  U[U^.  (x"’")]|j-l |c|} 


fCu^Cx**)]  < min  l-l}  , 

It  is  obvious  from  the  code  construction  that  the  resulting  code  is  a 
D-bounded  distortion  code  and  therefore 


d(B^.9)  ^ D 


for  all  9 € A.  Also,  since 


min  » 


we  have 


?(B^.e)  - (nk)‘^EQ{mln  i[U^^(x"‘^)] } 


< (nk)'^tnin  Eg  -^Cu^^Cx"*")] 


min  n"  Eg  £[u^(x")]  + (nk)‘^j'log2lc|] 


< r (n,D,0)  + e/2 


for  all  9 € A.  Using  (83)  completes  the  proof. 

Proof  of  Theorem  7.  The  transition  matrix  of  a binary  first-order  Markov 
source  has  the  form 


1 - q 


where  0 ^ p,  q ^ 1.  li  p and  q are  not  both  0,  the  corresponding  vector  of 


stationary  probabilities  is 


n - 

- \p+q  p+q/ 


The  source  (A^,jig*)  given  by 


1 0 


0 1 


is  nonergodic  and  H can  be  any  probability  vector. 


m 
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As  in  [ll] , we  define  the  matrix  metric  | 6^  - 82I  between  sources 

Hg  - and  by 

®1  ^^2 

|e^  - e^l  -maxfjn^^^  - - p^j^I.  i ^ i,  j s 2}. 

Under  the  matrix  metric,  A Is  compact  and  for  any  n both  R„  (D)  and 

V ytl 

r^(n,D,9)  are  continuous  functions  on  A since  both  depend  continuously  on 
the  n-th  order  distribution  of  which  Is  In  turn  a continuous  function  of  6. 
As  In  [ll],  we  can  then  show  Rq(U)  Is  continuous  on  A.  Since 
r'^CD.Q*)  ■ R0*(D)  ■ 0 and  since  by  Proposition  1,  r'^(D,0)  - ^3(0)  for  all 
other  6 € A,  r^(D,6)  Is  continuous  on  A.  As  In  [ll],  we  can  use  the 
subaddltlvlty  of  the  continuous  function  r^(n,D,6)  to  show  that  r^(n,D,6) 
converges  uniformly  to  the  continuous  function  r^(D,6)  on  A.  By  Proposition  2 
this  Implies  strongly  universal  codes  exist. 

Proof  of  Proposition  3.  Let  6 > 0.  Since  q(*)|n  ^1*  Q € A]  is  a family 
of  convex  strictly  decreasing  functions  of  the  argument  bounded  above  by 
InJ  and  below  by  0,  there  Is  an  e ' >0  such  that  for  any  e , 0 £ e ^ e ' , 


Rj,  g(D)  « R„^e(D  + «)  + 6 


(89 


for  all  N 2 1,  6 C A.  By  assumption,  there  Is  an  Integer  n and  a binary 


variable  rate  code  on  blocks  of  length  n,  such  that 


(90 


; (0)  < (ln2)"^[Rg(D)  + e] 


(91 


i 

I 

I 
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for  all  0 t A.  The  converse  coding  theorem  applied  to  (90)  gives 

(ln2)"^R^^g(D  + e)  ^ ?^(9)  (92) 

for  all  9 € A.  Combining  (89),  (91),  and  (92)  gives 

Rn^e(D)  ^ Rq(D)  + e + 6 

for  all  6 € A.  Since  6 and  e can  be  taken  arbitrarily  small,  the  theorem 
is  established. 

Proof  of  Proposition  4.  Fix 

D € (0,mlni'j"^  Z ^ p(x,y)jx  € A^]). 

v C A 
y - “(j 

For  any  positive  integer  n and  any  p,  0 •-  p < 1,  consider  the  n-th  order 
Markov  source  with  alphabet  and  transition  probability 

\ = \-n 
otherwise 

It  can  be  shown  that  the  stationary  distribution  for  this  transition  probability 
is 

p(x")  » J " (93) 

for  all  x"  € A«.  Let  (A/?,^i/^/  \)  he  the  stationary  ergodic  source 

*"  o utn,p; 

corresponding  to  this  transition  probability  and  stationary  distribution. 

Wa  have 

R0(n^p)(O)  • “P1"P  ■ J(l-p)ln(l-p)  (94) 


P(\IVl***“Vn>  “I  P 

' He 
i J-1 
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Let  be  the  meinoryless  source  which  assigns  probability  mass 


-1 


J to  each  x € Aq,  Since  Rg^(D)  > 0,  using  (94)  we  can  choose  p small  enough 


such  that  p)^®^  implying 

"ecn.p)"”  ^ 

But  from  (93),  R„  0(„  U)(»)  ■ Rq*(D)  *9  f*>at 

"■,,B(n.p)<“>  - '9(n,p)fl»  ^ 

Since  0(n,p)  € A for  all  n 2 1,  we  have  shown  that  {r  «(D)}  does  not 
converge  uniformly  on  A which  by  Proposition  3 implies  that  A is  not  strongly 
universally  encodable. 

Proof  of  Proposition  5.  We  use  proof  by  contradiction.  Fix  e > 0,  D s 0,  and 

assume  a binary  variable  rate  code  on  blocks  of  length  n,  ®>  (B,U,V), 

has  been  found  for  which 


and 


p^(9)  s D + e 


?^(0)  s (ln2)"^[Rg(D)  + c] 


(95) 

(96) 


for  all  0 € A.  Select  a countable  set  [a^,a2,.»*}  ^ A^  for  which 

p(a.  ,a.)  > 2(D  + e)  for  1 # j.  For  all  positive  integers  k,  let  ) 

^ n 

be  the  source  satisfying  p,g  [ , . . ,aj^,aj^  .a^^, . , ■ 1,  and  let  a^^  be  the 

k 

n-tuple  of 

Claim  . U(a?)  I*  U(a?)  for  1 

j 

Proof  of  Claim.  If  U(«J)  ■ U(«j)»  then 

vCu(aJ)]  - v[U(aJ)]  . 


(97) 
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